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administering help. Unfortunately, the quality of these sites is not 
regulated and therefore varies widely. An author-devised rating scale was 
used to assess six dimensions of site quality, including accuracy, 
practicality, normalization, sense of belonging, referral, and feedback 

mechanisms- — The-present investigation provides descriptive information about 

the nature of the sources evaluated. It also permits a comparison of the 
relative quality of the available sites across three different content 
domains: the needs of parents, substance abuse problems, and emotional 
disorders. A total of 210 web sites which included three different 
psychological problem categories (autism, bipolar disorder, and depression) 
were studied. Availability of quality resources and prevalence of the 
problems being targeted was not positively associated. This study helps to 
specify areas of unmet need and may assist in the development of additional 
resources on the Internet. The rating scale is appended. (Author/EMK) 
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Abstract 



The development of the Internet has provided an 
entirely new means of obtaining psychological healthcare, 
web sites have been set up for the purpose of providing 
information about psychological problems and administering 
help. Unfortunately, the quality of these sites is not 
regulated and therefore varies widely. An author-devised 
rating scale was used to assess six dimensions of site 
quality, including accuracy, practicality, normalization, 
sense of belonging, referral, and feedback mechanisms. The 
present investigation provides descriptive information about 
the nature of the sources evaluated. It also permits a 
comparison of the relative quality of the available sites 
across three different content domains. These domains 
include the needs of parents, substance abuse problems, and 
emotional disorders. This study helps to specify areas of 
unmet need and may assist in the development of additional 
resources on the Internet . 
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Introduction 



The Internet offers a myriad of opportunities to 
improve delivery of mental healthcare and enhance the lives 
of those affected by brain and behavioral disorders. The 
new electronic information technologies are fostering 
revolutions in how direct services are provided, and also 
facilitate optimal care indirectly, by keeping professionals 
more up-to-date and allowing larger-scale research on 
treatment effectiveness. 

One example of the utility of computer technologies in 
improving the delivery of treatment is the Newark Target 
Cities Project. This project was undertaken in order to 
unify the fragmented substance abuse treatment programs in 
Newark, New Jersey and thus eliminate the inefficiency 
inherent in a divided system. This project created a 
network of treatment providers, provided a standard means of 
assessing and placing clients, and allowed for clients to be 
matched with the most appropriate treatment program via a 
computerized management information system (Kraft & 
Dickinson, 1997) . 

The major contribution of this system was that it 
allowed for a treatment provider to access clients’ complete 
case histories. Prior to the implementation of this system 
it was difficult for one treatment center to discover what 
sorts of treatments a client had previously undergone. 
Resources and energy were wasted on repeating entrance 



interviews and treatment programs that may have already 
failed. This program effectively minimizes this sort of 
duplication waste by providing a complete record of a client 
across every treatment center (Kraft & Dickinson, 1997) . 

This is merely one example of the positive effects of 
one relatively small network. Its easy to see what a global 
network such as the Internet can do to enhance treatment. 

It has allowed for collaborative peer consultation in the 
form of professional user lists. Chambliss (1996) 
illustrates the utility of these lists in providing timely 
information about how to proceed with relatively rare 
situations . 

Innovations in direct service delivery include the 
development of problem-specific lists that create de facto 
support groups, expert -mediated Web sites that offer 
everything from virtual milieu therapy to parenting advice 
and electronic individual psychotherapy with therapists a 
nation away. For consumers who are introverted, homebound, 
or for when privacy is a priority, these remote forms of 
treatment are extremely attractive. 

For those with the requisite cognitive and computer 
skills and the access to appropriate equipment, the Internet 
literally opens up a world of treatment possibilities 
previously unimaginable. The result is a growing optimism 
among treatment professionals concerning the future 
possibilities for the Internet. 

Sampson et al . . (1997) envision the future "information 
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highway" as providing clients in remote locations with 
access to a variety of specialists that would otherwise be 
unavailable. They predict that Internet therapy will 
consist of a combination of counseling sessions by means of 
video conferencing and computer assisted instruction. 

Unfortunately the use of the Internet does not only 
open up new possibilities; it also creates new problems. In 
fact, the interaction of counselors and clients by means of 
electronic media presents several legal and ethical 
problems. One major problem concerns the issue of 
confidentiality. Although the use of e-mail seems to be 
very private in nature, it is arguable that privacy in this 
form of communication is an illusion. The reason for this 
is that after messages are sent they commonly go to a large 
storage facility like that of a university. Once there they 
are fair game to any hacker who might be so inclined to 
peruse someone else's mail (Shapiro & Schulman, 1996). 

Another problem with e-mail therapy results from the 
succinct and sporadic nature of the interaction between 
client and counselor. Healthcare providers in general do 
not condone the use of e-mail treatment as a plausible 
replacement for face-to-face counseling and indeed provide 
disclaimers that state this. The counselors assume that no 
professional relationship exists, but this can be subject to 
gross misinterpretation on the part of the client (Shapiro & 
Schulman, 1996) . 

Regardless of these problems, the use of the Internet 
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as a supplement to traditional methods of counseling is a 
worthwhile endeavor. If counseling is defined as “ a 
learning process designed to help people learn more 
effective ways of coping with their emotional, social, and 
career problems” (Sampson & Krumboltz, 1991) , then the use 
of the Internet as a means of disseminating information is 
valid. 

Research studies have already shown that computer 
mediated networking can be a valuable tool in education. One 
example of this is a study done by Kruger et al . . (1996) . 

They used a combination of face-to-face meetings and 
computer mediated communication to extend training in team 
problem solving. Though the subjects did not perceive the 
computer -mediated communication to be as important as the 
face-to-face meetings, they did view it as important to 
their mastery of the training. Based on these findings, 
Kruger et al . . (1996) concluded that the Internet has 

substantial promise for enhancing traditional training. 

In April 1996, Sampson, Kolodinsky, and Greeno (1997) 
sought to determine the prevalence of these resources on the 
Internet. Using the WebCrawler search engine they conducted 
a search of the word 'counseling.' The results of this 
search showed the existence of 3,764 home pages containing 
that term. Further analysis of these pages showed that 45% 
of these sites pertained to psychological counseling and the 
rest were either not accessible or merely contained the word 
counseling somewhere in their text. 
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Sampson et al . . repeated this search with the same 
search engine only three months later and discovered that 
the number of home pages had grown to 4,584. That is a 15% 
increase in three months. It is virtually impossible for 
both consumers and professionals to keep up with this rapid 
growth . 

Much of the existing literature pertaining to the 
Internet within the field of psychology is descriptive in 
nature. The main focus of journal articles has been placed 
on the problems and technological limitations associated 
with the current Internet (eg. Sampson et al . . 1997; Allen & 
Kostenbader, 1995; Frisse et al . . 1994) and projections 
about its eventual utility (eg. Huang & Alessi, 1996; 

Sampson et al . . 1997) . The subject of the Internet as it is 
currently employed is given little attention. 

The Internet is a useful and accessible tool for 
providing mental health care; however, specific goals and 
standards for its use must be met. Managed care 
organizations are already investigating means for reducing 
costs through computerized medical education, and such 
forces will further encourage the development of networked 
information resources (Huang & Alessi, 1996) . By 
anticipating the potential abuses of the Internet, 
professional associations can assure that the information 
highway helps rather than harms clients. 

Although it would be beneficial for clients to be as 
knowledgeable about their disorders as their therapists, 




8 



inaccurate and inappropriate information could be 
detrimental to the outcome of their treatment. The validity 
of data delivered via computer networks needs to be screened 
prior to patient and public exposure (Sampson, Kolodinsky & 
Greeno, 1997) . Additionally, counselors need to be educated 
and trained in administering this treatment. 

The recent explosion of behavioral healthcare resources 
on the Net has left many consumers and professionals 
overwhelmed. The Quality of Websites and lists is very 
uneven and no systematic mechanism exists to evaluate the 
utility of a given resource efficiently. Jacobson and Cohen 
(1997) have discussed the importance of teaching students to 
evaluate the quality of Internet sites. They argue that 
users of sites should consider the dimensions of accuracy, 
comprehensiveness, currency, availability of hyperlinks, and 
the Web sites’ style and functionality. The current study 
examined both the quantity and quality of extant behavioral 
healthcare resources on the Internet for a wide variety of 
problems. Sites and lists were assessed in terms of criteria 
developed by the authors, reflecting the objectives of ideal 
electronic resources. 

Doran et al . (in press) used this same paradigm to 
evaluate Internet resources that pertained to common 
parenting problems, substance abuse problems, anxiety 
disorders, and eating disorders. The results of this study 
seemingly indicated that a positive relationship existed 
between the quality of Web sites and the prevalence of their 



topics. In other words, sites that pertained to relatively 
common problems received better ratings than sites that 
pertained to less common problems . 

This study was designed to expand upon the former 
study. It sought to evaluate Internet resources that 
pertain to autism, bipolar disorder, and depression. It 
utilized the same criteria as in the previous study such 
that the data from both studies were strictly comparable. 

This study provides comparative data describing the 
quantity and quality of Internet services currently 
available to those affected by different disorders. This 
summary may be useful to those interested in specifying 
areas of unmet needs. It may also help those seeking 
exemplary models of how this new medium can be effectively 
applied. 



"The democratization of knowledge is all to 
the good, if that means the democratization of 
access to knowledge. But democratization of the 
access to knowledge should not be confused with 
the democratization of knowledge itself. And this 
is where the Internet, or any system of electronic 
networking, may be misleading or even pernicious. 
In cyberspace, every source seems as authoritative 
as every other. 

The Internet is an equal opportunity 
resource; it recognizes no rank or status or 
privilege. In that democratic universe, all 
sources, all ideas, all theories seem equally 
valid and pertinent. It takes a discriminating 
mind, a mind that is already stocked with 
knowledge and trained in critical discernment, to 
distinguish between Peanuts and Shakespeare -- 
between the trivial and the important, the 
ephemeral and the enduring, the true and the 
false" (Himmelfarb, 1997) . 



Autism 



Autism is a fairly common problem. Autistic disorder 
effects approximately 2-5 people out of every 10,000 (APA, 
1994) . It is a neurological disorder characterized by a 
marked impairment of social, emotional and intellectual 
functioning. The basic criteria for the diagnosis of 
Autistic disorder include abnormally low levels of social 
development, impairment of both verbal and non-verbal 
communication skills, abnormal responses to stimuli, and 
behavior marked by its repetitive nature (Edelson, 1997) . 

Autistic disorder is typically diagnosed in early 
childhood when the individuals’ diminished social 
development begins to become apparent. Seventy five percent 
of those individuals diagnosed with Autistic Disorder are 
also diagnosed with mental retardation, IQ between 35-50 
(APA, 1994) . 

The causes of autism are not known, but various 
theories as to its causes have been stated (Edelson, 1997) . 
These theories are of various natures. They include 
psychogenet ic, psychosocial, cognitive (Wolf -Schein, 1996), 
and biogenetic theories (Reichelt et al . . 1986) . All of 
these theories seemingly place the responsibility for the 
development of this disorder, to some extent, upon the 
parents. Responsibility falls upon the parents either as a 
result of their genes or the environment that they provided 
for the affected child. 
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It is therefore easy to conceive that parents of 
Autistic children would feel some level of responsibility 
and guilt for their children’s condition. These feelings 
are potentially elevated by typical autistic behaviors such 
as failure to cuddle, lack of eye contact, and failure to 
respond to parents’ voices or gestures (APA, 1994) . Thus, 
it becomes apparent that the autistic individuals are not 
the only ones who are in need of assistance. 

Parents of autistic individuals may then be driven to 
find various sources of information, support, and 
normalization in attempts to understand their child’ s 
condition and abate their own feelings of guilt. The 
Internet provides an ideal way for these parents to 
accomplish this. However, in order to be truly ideal, the 
sites that service these individuals must be high in 
quality. It is the goal of this research to differentiate 
the high quality sites from those that are lacking in 
quality. 



Bipolar Disorders 

Bipolar disorders consist of several subcategories that 
describe specific aspects of diagnoses. These categories 
include single manic episode, most recent episode hypomanic, 
most recent episode manic, most recent episode mixed, most 
recent episode depressed, most recent episode unspecified, 
and recurrent major depressive episodes with hypomania. 



Generally, bipolar disorders are marked by episodes of mania 
or hypomania that are typically accompanied by episodes of 
depression (Sue et al . , 1994) . 

Bipolar disorders are fairly common. Although there is 
some evidence suggesting that clinicians have a tendency to 
overdiagnose Schizophrenia rather than bipolar disorder in 
young people and some ethnic groups, the incidence rates for 
bipolar disorders do not seem to differ across gender, race, 
or ethnicity (APA, 1994) . Kessler et al . (1994 cited in 

Nolen-Hoeksema, 1998) estimate that 1-2 out of 100 people 
will experience at least one bipolar episode over the course 
of their lives. 

Approximately 90 percent of those diagnosed with a 
bipolar disorder have more than one episode in their 
lifetimes (APA, 1994) . The frequency, number, and duration 
of these episodes are largely dependent upon the bipolar 
individual. Episodes may last for one day or as long as 
several months. Generally these episodes become 
increasingly frequent and closer together with the passage 
of time (Nolen-Hoeksema, 1998) . 

Individuals diagnosed with bipolar disorder are thus 
left wondering when the next episode will occur. It is not 
uncommon for them to feel as though they can not trust their 
emotions or self -perceptions (Nolen-Hoeksema, 1998) . This 
may be a highly discomforting and isolating frame of mind. 

It is easy to see that a means of social support and 
normalizing information could aid individuals with bipolar 



disorder . 



High quality web sites would be able to provide these 
individuals with 24 -hour access to this kind of information. 
Those with the requisite hardware could easily participate 
in on-line support groups, read case histories, learn about 
particular aspects of their disorder, and find referrals to 
other avenues of obtaining help. However, in order for web 
sites to be a useful part of recovery, the sites that are 
high in quality must be differentiated from those that are 
low in quality. 



Depression 

Depression is a serious condition that can be 
emotionally debilitating. In clinical terms depression is 
described as a psychoneurotic or psychotic disorder which 
particularly includes sadness, inactivity, thinking and 
concentration difficulty, a significant increase or decrease 
in appetite and sleep, feelings of dejection and 
hopelessness, and in some cases suicidal tendencies 
(Webster, 1994) . Being confronted by these seemingly 
lifeless symptoms can easily be seen as a terrible way to 
live one’ s life. 

Unfortunately this distressing disorder is rather 
common. Although women experience depression about twice as 
often as men do (Culbertson, 1997) , the risk for men is 
relatively high also. The lifetime risk for major 
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depressive disorder in community samples ranges from 10% to 
25% for women and from 5% to 12% for men (APA, 1994) . 

Because depression is both unpleasant and frequent within 
the population, it is crucial for individuals to have access 
to information about the disorder and treatment options. 

Within the recent past, those afflicted with depression 
would typically have to independently locate a therapist in 
order to obtain information and treatment for their 
depression. Today, however, there is an additional option 
in the Internet . 

The Internet has recently become a prolific source of 
psychological healthcare information. One who is afflicted 
with depression can learn about his or her disorder, obtain 
the names of therapists, find some suggestions for self- 
help, find peer support, and in some cases even find 
treatment on the web. 

Although the Internet is not a substitute for 
traditional forms of treatment, it is a viable means of 
complementing those treatments. To ensure that the Internet 
becomes a treatment aid that is high in quality, it is 
necessary for its psychological resources to undergo an 
evaluation. it is the goal of this evaluation process to 
distinguish between sites of high and low quality such that 
the utility of the Internet as a treatment tool may be 
maximized. 



Method 



Source and researchers 



A total of 210 web sites were evaluated from three 
different psychological problem categories which included 
autism, bipolar disorder, and depression. The sites were 
chosen randomly from various common search engines on the 
Internet. Six trained undergraduate psychology majors from 
a small liberal arts college on the East Coast completed the 
evaluations . 

After a period of training in the psychological problem 
areas and on the evaluation process, each rater evaluated 
roughly 25-40 sites pertaining to one of the three 
categories. After the initial ratings, a sample of randomly 
chosen sites was reassigned to a second evaluator in order 
to permit assessment of interrater reliability. 

Materials 

Several computers linked to the Internet served as 
tools for access to the Internet. A standardized evaluation 
form assessing 6 separate dimensions was used to record the 
ratings for each site along with some demographic 
information. Evaluators were asked to rate how well each 
site provided the following: l) clear and accurate 
information 2) "how-to" suggestions for change: practical 
exercises 3) destigmatizing information; promotion of 
normalization 4) promotion of a sense of belonging; 
information to help combat loneliness 5) referral 
mechanisms, if users found that additional help is needed 
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and 6) outcomes assessment; feedback mechanism. 



Results 

Scores on the 6 dimensions were totaled for each site, 
yielding a summary measure of each site's quality. A one- 
way ANOVA was performed to compare the summary quality 
scores across the problem topics. Significant differences 
(pc.OOl) emerged across the problem topics. Ratings for 
sites pertaining to depression received significantly more 
negative evaluations than sites for the other topics (see 
Table l) . 

Overall, the ratings again suggested that the average 
quality of the current sites was not very impressive. The 
mean rating across problem topics for all site dimensions 
was 1.92 (s.d.=l.iO) on the 0-4 point scale where 0=absent, 
i=extremely inadequate, 2=somewhat inadequate, 3=somewhat 
adequate, and 4=outstanding . 

Inter-item correlations revealed significant 
relationships among all items used to rate the resources. 
However, the correlations were not so high as to suggest 
that any items failed to make a meaningful contribution to 
the overall rating (see Table 2) . 

When the data from this study were combined with the 
data from the previous study (Doran et al . . in press) , three 



problem domains emerged. These problem domains were the 
needs of parents, substance abuse problems, and emotional 
disorders. Included in the “needs of parents" domain were 
sites pertaining to common parenting problems as well as 
sites pertaining to autism. The substance abuse domain 
consisted of sites addressing Alcohol abuse, drug abuse, and 
eating disorders. The emotional disorder domain consisted 
of anxiety disorders, depression, and bipolar disorders. 

A oneway ANOVA was conducted in order to assess the 
differential quality of Internet resources across both 
studies. The average quality of sites addressing the needs 
of parents was significantly higher than the quality of 
sites directed at the other two types of problems (x = 

14.83; sd = 5.06; n = 215) . Next highest were ratings of 
sites addressing substance use problems (x = 13.65; sd = 
6.28; n = 166) . Both of these problem domains had sites 
rated significantly higher than those targeting emotional 
disorders (F = 30.86; pc.OOl; df = 2/512). The lowest 
ratings were given to the sites directed at individuals with 
Emotional Disorders (x = 10.44; sd = 6.27; n = 200) . 

In order to enhance consistency across raters, 
additional training was conducted prior to the data 
collection for this study. This training was associated 
with significant improvement in the consistency of the 
evaluation technique. Interrater reliability for the data 
from this study was .85 (p<.01), and the overall reliability 
for all cases, including the previous study, was .75 
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(p< . 01) . 



Discussion 

The results of this study were not consistent with our 
interpretation of the previous study’s results. These data 
did not indicate the positive association between the 
availability of quality resources and prevalence of the 
problems being targeted that was previously observed. in 
fact, a converse relationship was observed in that those 
sites pertaining to the most prevalent problem, depression, 
received the worst quality ratings rather than the best. 

The availability of higher quality resources for those 
interested in Autism and Bipolar disorders may actually be 
the result of depression having both high rates of 
prevalence and social acceptability. it may be that since 
depression is relatively common and does not bear any severe 
social stigma, more computer users are willing to post web 
pages on this topic than on other, more socially ostracized 
topics such as autism and bipolar disorder. Thus a 
disproportionate number of poorly developed personal home 
pages and product -marketing sites on the web may have caused 
the quality ratings for depression sites to be low. 

General 

Combining the data from this and the previous study 
revealed significant differences among the three general 
topic domains in terms of site quality. The results show 
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that sites that service the needs of parents are 
consistently higher in quality than those that are aimed at 
substance abuse problems, which in turn are better than 
sites that address emotional disorders. 

One possible explanation for this may be that parents 
simply crave more information than those who are not 
parents; creating a greater market for Internet resources 
aimed at the needs of parents. This makes intuitive sense 
in that parents are seemingly under a great deal of pressure 
to perform as “good parents” . As a result, parents seek out 
information that can improve their parenting skills, 
increase their confidence in what skills they already 
possess, and show them that they are not alone in their 
self-doubt. High quality Internet resources have the 
potential to address all of these needs (at least 
partially) . The high demand for such resources may drive 
their creation. 

The intended audience for the parenting sites evaluated 
in this study were either parents of autistic children or 
parents whose children presented them with more typical 
challenges. In all cases, the parents were presumed to be 
normally functioning adults. In contrast, the substance 
abuse and emotional disorder sites were aimed primarily at 
those suffering from one kind of disorder. It may be that 
creators of the latter sites held themselves to a lower 
standard of excellence, because of negative attitudes toward 
the intended audience. 



:/ 



Extensions of this study should include additional 
training of raters in the consistent use of the rating 
system in order to further improve interrater reliability. 
Randomized assignment of cases to raters would further 
reduce the problems of confounding, although this would 
require that all raters become expert in all problem areas, 
inclusion of behavioral healthcare consumers in the panel of 
raters would also enhance the usefulness of this research. 
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Table l 

Mean Quality Rating by Topic 





Number 

of 

Cases 


Mean 

Quality 

Rating 


Std. 

Deviation 


Autism 


75 


14 .09 


5.43 


Bipolar 

Disorder 


58 


12.36 


7.32 


Depression 


77 


8.36 


5 . 77 


Total 


210 


11 .51 


6 . 59 
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Inter-Item Correlations 
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Psych Help Resources 
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Rater 

Disorder 

Site Name 

Site Address 

Type of Site: General Disorder Class Specific 
Disorder Specific Other 

Site Motive: Informative Provide Help 

Commercial Other 

Use a four point scale to evaluate the resource on each of 
the following dimensions. 

0 Absent 

1 Extremely Inadequate 

2 Somewhat inadequate 

3 Somewhat Adequate 

4 Outstanding 

Clear and Adequate Information 

Practical Suggestions for change 

Dest igmatizing Information; Promotion of Normalization 

Promote sense of Belonging; Combat Loneliness 

Referral Mechanisms 

Outcomes Assessment; Feedback Mechanisms 
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Internet Project Spring ’ 98 



Evaluation Descriptions 

Type of Site: example : general = “mental health site", 

disorder class specific = “chemical dependency", disorder 
specific = “alcoholism”, other = written explanation. 

Site Motive: Informative = purely passing on 

information. Provide Help = suggestions for change, self- 
help, Commercial = goal to obtain profit. Other = written 
description . 

Clear and Adequate Information = Page is well written and easy 
to follow, no apparent gaps in information. 

Practical Suggestions for Change = How to’ s , 

Destigmatizing information; Promotion of Normalization = 
Personal accounts, success stories, inspiration. 

Promote sense of belonging, combat loneliness = chat lines, 
on-line support groups, immediate access to others. 

Referral Mechanisms = contacts to community groups, and other 
Internet sites (non-interactive) Non cyber referrals. 

Outcomes Assessment; Feedback Mechanisms = counseling via the 
Internet only (e-mail or one on one chats with some one in a 
counseling position) Users access to information on own 
recovery rates. 
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